Support Unique Conversation ID by jgieringer · Pull Request #104 · SpringCare/VERA-MH

jgieringer · 2026-02-10T00:25:44Z

Description

Conversation ID support and naming

Naming: Renamed the run-level ordinal from conversation_id to conversation_index in the runner and tests. It remains the 1-based index of a conversation within a run; the result dict still uses the key "id" for compatibility (but noted to revisit to change to "index".
Conversation ID (per-conversation): Added support for a unique, per-conversation identifier used by LLM clients (e.g. for API thread/session tracking):
- LLMInterface: conversation_id, create_conversation_id(), and _update_conversation_id_from_metadata().
  - LLMInterface sets self.conversation_id at init.
  - For any subclass needing a conversation_id, it is available to put in request & response metadata
    - at the end of generate_response(), call _update_conversation_id_from_metadata() and overwrite if API returns with new conversation_id.
- Runner: To support setting a new conversation_id (and starting a new conversation), a new agent is created per persona created.
  - This is refactored from only one agent being created per Runner object
    - If this is an unsatisfactory refactor, I will need to look into generating a new conversation_id when a new persona "enters the room"
- Simulator: No conversation_id handling here, this is only at the llm_client level
Response metadata: Replaced the get_last_response_metadata() API with a last_response_metadata property (backed by _last_response_metadata) so reading it always returns a deep copy. Implementations that mutate metadata in place use _last_response_metadata; the setter still accepts full dicts via self.last_response_metadata = {...}.
Docs: Updated docs/evaluating.md with a “Conversation flow and history” section and clarified how to use last_response_metadata, conversation_id, and _update_conversation_id_from_metadata() when implementing a custom LLM client.

Copilot

Pull request overview

Adds first-class support for a per-conversation unique identifier (conversation_id) on LLMInterface, while also clarifying/renaming the runner’s run-level ordinal to conversation_index and standardizing response metadata access via a last_response_metadata property.

Changes:

Introduces LLMInterface.conversation_id, plus create_conversation_id() / ensure_conversation_id(), and converts response metadata access from get_last_response_metadata() to last_response_metadata.
Updates built-in LLM clients + tests to use the new metadata property and to call ensure_conversation_id() after responses/errors.
Renames runner/test parameter conversation_id ➜ conversation_index while keeping result key "id" for compatibility; updates docs accordingly.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`llm_clients/llm_interface.py`	Adds `conversation_id` + metadata property and helper methods.
`llm_clients/openai_llm.py`	Switches to `_last_response_metadata` for in-place updates; calls `ensure_conversation_id()`.
`llm_clients/azure_llm.py`	Same as above for Azure client, including error paths.
`llm_clients/claude_llm.py`	Same as above for Claude client.
`llm_clients/gemini_llm.py`	Same as above for Gemini client.
`llm_clients/ollama_llm.py`	Calls `ensure_conversation_id()` and removes old metadata getter.
`generate_conversations/runner.py`	Renames `conversation_id` ➜ `conversation_index`; resets `agent.conversation_id` per conversation.
`generate_conversations/conversation_simulator.py`	Uses `last_response_metadata` property for logging metadata.
`tests/unit/llm_clients/test_*`	Updates tests to use `last_response_metadata` and renames copy-behavior tests.
`tests/unit/llm_clients/test_llm_interface.py`	Adds unit coverage for `conversation_id` helpers.
`tests/mocks/mock_llm.py`	Removes old metadata getter; calls `ensure_conversation_id()`.
`tests/integration/test_conversation_runner.py`	Renames `conversation_id` ➜ `conversation_index` in integration tests.
`docs/evaluating.md`	Documents the new metadata property and `conversation_id` flow.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

generate_conversations/runner.py

llm_clients/llm_interface.py

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/evaluating.md

llm_clients/openai_llm.py

llm_clients/gemini_llm.py

llm_clients/claude_llm.py

llm_clients/azure_llm.py

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

generate_conversations/runner.py

tests/integration/test_conversation_runner.py

generate_conversations/runner.py

docs/evaluating.md

generate_conversations/runner.py

emily-vanark · 2026-02-11T22:33:19Z

llm_clients/azure_llm.py

                    usage = metadata["token_usage"]
-                    self.last_response_metadata["usage"] = {
+                    self._last_response_metadata["usage"] = {
                        "input_tokens": usage.get("input_tokens", 0),


this is off-topic, but... do we save this metadata anywhere? It doesn't seem to be in the logging output for chat generation, or the logs output for judging... but if we have total token usage for each conversation and judging evaluation somewhere that we could write out, it would help... everyone with understanding costs. (This is probably a separate ticket... but only if we really aren't storing it anywhere.)

👍 https://springhealth.atlassian.net/browse/SAF-203

emily-vanark

Couple questions / suggestions, but the tests run, and it looks okay to me. Would like @sator-labs ' opinion on it if he has time.

nz-1

lgtm - left one comment for consideration

generate_conversations/runner.py

sator-labs

LGTM

docs/evaluating.md

generate_conversations/runner.py

sator-labs · 2026-02-12T02:36:42Z

generate_conversations/runner.py

+            model_name=self.agent_model_config["model"],
+            name=self.agent_model_config.get("name", "Provider"),
+            system_prompt=self.agent_model_config.get(
+                "system_prompt", "You are a helpful AI assistant."


I would maybe have this as a optional arg with the default, since it seems a potentially consequential decision and it's buried here

generate_conversations/runner.py

llm_clients/llm_interface.py

tests/unit/llm_clients/test_helpers.py

conversation_id -> conversation_index + support unique conversation_id

97af907

jgieringer requested a review from Copilot February 10, 2026 00:25

Copilot started reviewing on behalf of jgieringer February 10, 2026 00:26 View session

Copilot AI reviewed Feb 10, 2026

View reviewed changes

generate_conversations/runner.py Outdated Show resolved Hide resolved

llm_clients/llm_interface.py Outdated Show resolved Hide resolved

jgieringer added 4 commits February 10, 2026 12:04

prevent race condition when resetting agent convo id

6fb999e

deep copy for last_response_metadata

63a861e

Merge branch 'main' into jgieringer/convo-id

4c87bba

assign conversation_id at init

cb4fef7

jgieringer marked this pull request as ready for review February 10, 2026 23:54

jgieringer requested review from Copilot, emily-vanark, nz-1 and sator-labs February 10, 2026 23:54

Copilot started reviewing on behalf of jgieringer February 10, 2026 23:54 View session

Copilot AI reviewed Feb 10, 2026

View reviewed changes

jgieringer mentioned this pull request Feb 11, 2026

Support Custom API #105

Merged

clarity

eabfb6d

jgieringer requested a review from Copilot February 11, 2026 00:14

Copilot started reviewing on behalf of jgieringer February 11, 2026 00:14 View session

Copilot AI reviewed Feb 11, 2026

View reviewed changes

jgieringer added 4 commits February 11, 2026 10:15

refactor always true test

733aa28

refactor to test behavior

42320f6

correct the test llm creation order

14c8ed0

kwargs

d4debf2

jgieringer commented Feb 11, 2026

View reviewed changes

generate_conversations/runner.py Outdated Show resolved Hide resolved

emily-vanark reviewed Feb 11, 2026

View reviewed changes

docs/evaluating.md Outdated Show resolved Hide resolved

emily-vanark reviewed Feb 11, 2026

View reviewed changes

generate_conversations/runner.py Show resolved Hide resolved

emily-vanark reviewed Feb 11, 2026

View reviewed changes

generate_conversations/runner.py Show resolved Hide resolved

emily-vanark reviewed Feb 11, 2026

View reviewed changes

emily-vanark approved these changes Feb 11, 2026

View reviewed changes

nz-1 approved these changes Feb 11, 2026

View reviewed changes

generate_conversations/runner.py Show resolved Hide resolved

sator-labs approved these changes Feb 12, 2026

View reviewed changes

jgieringer added 5 commits February 12, 2026 09:33

conversation > thread

feedd85

clean up doc

80fb5c2

comment for accepted failure

2ee3051

id -> index + documentation

5c5b6f4

test uuid

7c03595

jgieringer merged commit 13cdf16 into main Feb 12, 2026

jgieringer deleted the jgieringer/convo-id branch February 12, 2026 23:38

Conversation

jgieringer commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emily-vanark Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgieringer Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

emily-vanark left a comment

Choose a reason for hiding this comment

Uh oh!

nz-1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sator-labs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sator-labs Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

jgieringer Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jgieringer commented Feb 10, 2026 •

edited

Loading

emily-vanark Feb 11, 2026 •

edited

Loading